Lecture 2: Subgradient Methods

In this lecture, we discuss first order methods for the minimization of convex functions. We focus almost exclusively on subgradient-based methods, which are essentially universally applicable for convex optimization problems, because they rely very little on the structure of the problem being solved. This leads to effective but slow algorithms in classical optimization problems, however, in large scale problems arising out of machine learning and statistical tasks, subgradient methods enjoy a number of (theoretical) optimality properties and have excellent practical performance.